{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Check GPU's Memory Capacity\n",
        "\n",
        "By running `nvidia-smi` command, you can find out the GPU's memory capacity on the current system. \n",
        "\n",
        "With the standard GPU instance(___T4___) which is free, you can run 7B and 13B models. With the premium GPU instance(___A100 40GB___) which is paid with the compute unit that you own, you can even run 30B model! Choose the instance at the menu `Runtime` -> `Change runtime type` -> `Hardware accelerator (GPU)` -> `GPU class (Standard or Premium)`"
      ],
      "metadata": {
        "id": "xf3pUNyVO3WS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!nvidia-smi"
      ],
      "metadata": {
        "id": "L2MoM27rfaKK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "37daa644-f021-4a55-c007-86eb96ccdf9b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sun Apr  9 17:17:36 2023       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   49C    P8    10W /  70W |      0MiB / 15360MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Clone the repository"
      ],
      "metadata": {
        "id": "N0MDD9TuPTfJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone --branch alpaca-lora-new https://github.com/WangRongsheng/ChatGenTitle.git"
      ],
      "metadata": {
        "id": "a_i5DKBNnzAK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9612bb9a-997a-4267-810a-99af621092a3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'ChatGenTitle'...\n",
            "remote: Enumerating objects: 478, done.\u001b[K\n",
            "remote: Counting objects: 100% (189/189), done.\u001b[K\n",
            "remote: Compressing objects: 100% (136/136), done.\u001b[K\n",
            "remote: Total 478 (delta 125), reused 60 (delta 43), pack-reused 289\u001b[K\n",
            "Receiving objects: 100% (478/478), 2.80 MiB | 16.04 MiB/s, done.\n",
            "Resolving deltas: 100% (219/219), done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%cd ChatGenTitle"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Brj2nl57B1Zm",
        "outputId": "c5ac3e0a-6c2a-431a-ba4e-e6ceeccdb4d9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/ChatGenTitle\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Move into the directory of the cloned repository"
      ],
      "metadata": {
        "id": "HUuzxWGuPYLq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ls"
      ],
      "metadata": {
        "id": "wR-M8u7gsQqg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "503851a2-2c21-463d-e6fc-bf949c9fea62"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[0m\u001b[01;34mdata\u001b[0m/                            generate.py                   requirements.txt\n",
            "\u001b[01;34mdocs\u001b[0m/                            get_arxiv_multiprocessing.py  \u001b[01;34mtemplates\u001b[0m/\n",
            "export_hf_checkpoint.py          LICENSE                       \u001b[01;34mutils\u001b[0m/\n",
            "export_state_dict_checkpoint.py  README.md\n",
            "finetune.py                      requirements-all.txt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Install dependencies"
      ],
      "metadata": {
        "id": "XG8oy7BBPdMh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -r requirements.txt"
      ],
      "metadata": {
        "id": "moN-15x_ifHE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "384d73cd-8845-4bee-99e7-49f750d69adc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting git+https://github.com/huggingface/peft.git (from -r requirements.txt (line 9))\n",
            "  Cloning https://github.com/huggingface/peft.git to /tmp/pip-req-build-q07mxqux\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/peft.git /tmp/pip-req-build-q07mxqux\n",
            "  Resolved https://github.com/huggingface/peft.git to commit 1117d4772109a098787ce7fc297cb6cd641de6eb\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting git+https://github.com/huggingface/transformers.git (from -r requirements.txt (line 10))\n",
            "  Cloning https://github.com/huggingface/transformers.git to /tmp/pip-req-build-sjzym39u\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git /tmp/pip-req-build-sjzym39u\n",
            "  Resolved https://github.com/huggingface/transformers.git to commit 656e869a4523f6a0ce90b3aacbb05cc8fb5794bb\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting accelerate\n",
            "  Downloading accelerate-0.18.0-py3-none-any.whl (215 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m215.3/215.3 KB\u001b[0m \u001b[31m6.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: appdirs in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 2)) (1.4.4)\n",
            "Collecting loralib\n",
            "  Downloading loralib-0.1.1-py3-none-any.whl (8.8 kB)\n",
            "Collecting bitsandbytes\n",
            "  Downloading bitsandbytes-0.37.2-py3-none-any.whl (84.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.2/84.2 MB\u001b[0m \u001b[31m11.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting black\n",
            "  Downloading black-23.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m68.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting datasets\n",
            "  Downloading datasets-2.11.0-py3-none-any.whl (468 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m468.7/468.7 KB\u001b[0m \u001b[31m42.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting fire\n",
            "  Downloading fire-0.5.0.tar.gz (88 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m88.3/88.3 KB\u001b[0m \u001b[31m12.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.97-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m59.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting gradio\n",
            "  Downloading gradio-3.24.1-py3-none-any.whl (15.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.7/15.7 MB\u001b[0m \u001b[31m63.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.9/dist-packages (from accelerate->-r requirements.txt (line 1)) (2.0.0+cu118)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/dist-packages (from accelerate->-r requirements.txt (line 1)) (23.0)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.9/dist-packages (from accelerate->-r requirements.txt (line 1)) (5.9.4)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.9/dist-packages (from accelerate->-r requirements.txt (line 1)) (1.22.4)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.9/dist-packages (from accelerate->-r requirements.txt (line 1)) (6.0)\n",
            "Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.9/dist-packages (from black->-r requirements.txt (line 5)) (8.1.3)\n",
            "Collecting mypy-extensions>=0.4.3\n",
            "  Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
            "Collecting pathspec>=0.9.0\n",
            "  Downloading pathspec-0.11.1-py3-none-any.whl (29 kB)\n",
            "Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.9/dist-packages (from black->-r requirements.txt (line 5)) (2.0.1)\n",
            "Requirement already satisfied: typing-extensions>=3.10.0.0 in /usr/local/lib/python3.9/dist-packages (from black->-r requirements.txt (line 5)) (4.5.0)\n",
            "Requirement already satisfied: platformdirs>=2 in /usr/local/lib/python3.9/dist-packages (from black->-r requirements.txt (line 5)) (3.2.0)\n",
            "Requirement already satisfied: ipython>=7.8.0 in /usr/local/lib/python3.9/dist-packages (from black->-r requirements.txt (line 5)) (7.34.0)\n",
            "Collecting tokenize-rt>=3.2.0\n",
            "  Downloading tokenize_rt-5.0.0-py2.py3-none-any.whl (5.8 kB)\n",
            "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 7)) (2.27.1)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 7)) (2023.3.0)\n",
            "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 7)) (4.65.0)\n",
            "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 7)) (9.0.0)\n",
            "Collecting aiohttp\n",
            "  Downloading aiohttp-3.8.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m68.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting dill<0.3.7,>=0.3.0\n",
            "  Downloading dill-0.3.6-py3-none-any.whl (110 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.5/110.5 KB\u001b[0m \u001b[31m15.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting xxhash\n",
            "  Downloading xxhash-3.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.2/212.2 KB\u001b[0m \u001b[31m25.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting huggingface-hub<1.0.0,>=0.11.0\n",
            "  Downloading huggingface_hub-0.13.4-py3-none-any.whl (200 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m200.1/200.1 KB\u001b[0m \u001b[31m24.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting multiprocess\n",
            "  Downloading multiprocess-0.70.14-py39-none-any.whl (132 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m132.9/132.9 KB\u001b[0m \u001b[31m19.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting responses<0.19\n",
            "  Downloading responses-0.18.0-py3-none-any.whl (38 kB)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 7)) (1.4.4)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.9/dist-packages (from fire->-r requirements.txt (line 8)) (1.16.0)\n",
            "Requirement already satisfied: termcolor in /usr/local/lib/python3.9/dist-packages (from fire->-r requirements.txt (line 8)) (2.2.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from transformers==4.28.0.dev0->-r requirements.txt (line 10)) (3.10.7)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1\n",
            "  Downloading tokenizers-0.13.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m97.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from transformers==4.28.0.dev0->-r requirements.txt (line 10)) (2022.10.31)\n",
            "Collecting aiofiles\n",
            "  Downloading aiofiles-23.1.0-py3-none-any.whl (14 kB)\n",
            "Collecting gradio-client>=0.0.5\n",
            "  Downloading gradio_client-0.0.8-py3-none-any.whl (20 kB)\n",
            "Collecting orjson\n",
            "  Downloading orjson-3.8.10-cp39-cp39-manylinux_2_28_x86_64.whl (140 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m140.5/140.5 KB\u001b[0m \u001b[31m16.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting python-multipart\n",
            "  Downloading python_multipart-0.0.6-py3-none-any.whl (45 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.7/45.7 KB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpx\n",
            "  Downloading httpx-0.23.3-py3-none-any.whl (71 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m71.5/71.5 KB\u001b[0m \u001b[31m10.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pydantic in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 12)) (1.10.7)\n",
            "Requirement already satisfied: markdown-it-py[linkify]>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 12)) (2.2.0)\n",
            "Collecting semantic-version\n",
            "  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
            "Collecting uvicorn\n",
            "  Downloading uvicorn-0.21.1-py3-none-any.whl (57 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.8/57.8 KB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pillow in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 12)) (8.4.0)\n",
            "Collecting websockets>=10.0\n",
            "  Downloading websockets-11.0.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.5/129.5 KB\u001b[0m \u001b[31m14.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: markupsafe in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 12)) (2.1.2)\n",
            "Collecting mdit-py-plugins<=0.3.3\n",
            "  Downloading mdit_py_plugins-0.3.3-py3-none-any.whl (50 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.5/50.5 KB\u001b[0m \u001b[31m7.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 12)) (3.7.1)\n",
            "Collecting ffmpy\n",
            "  Downloading ffmpy-0.3.0.tar.gz (4.8 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: altair>=4.2.0 in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 12)) (4.2.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 12)) (3.1.2)\n",
            "Collecting fastapi\n",
            "  Downloading fastapi-0.95.0-py3-none-any.whl (57 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.1/57.1 KB\u001b[0m \u001b[31m8.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pydub\n",
            "  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.9/dist-packages (from altair>=4.2.0->gradio->-r requirements.txt (line 12)) (4.3.3)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.9/dist-packages (from altair>=4.2.0->gradio->-r requirements.txt (line 12)) (0.12.0)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.9/dist-packages (from altair>=4.2.0->gradio->-r requirements.txt (line 12)) (0.4)\n",
            "Collecting multidict<7.0,>=4.5\n",
            "  Downloading multidict-6.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.2/114.2 KB\u001b[0m \u001b[31m15.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting async-timeout<5.0,>=4.0.0a3\n",
            "  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets->-r requirements.txt (line 7)) (22.2.0)\n",
            "Collecting yarl<2.0,>=1.0\n",
            "  Downloading yarl-1.8.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m264.6/264.6 KB\u001b[0m \u001b[31m32.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting frozenlist>=1.1.1\n",
            "  Downloading frozenlist-1.3.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (158 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m158.8/158.8 KB\u001b[0m \u001b[31m21.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets->-r requirements.txt (line 7)) (2.0.12)\n",
            "Collecting aiosignal>=1.1.2\n",
            "  Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
            "Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.9/dist-packages (from ipython>=7.8.0->black->-r requirements.txt (line 5)) (5.7.1)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.9/dist-packages (from ipython>=7.8.0->black->-r requirements.txt (line 5)) (4.4.2)\n",
            "Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.9/dist-packages (from ipython>=7.8.0->black->-r requirements.txt (line 5)) (4.8.0)\n",
            "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from ipython>=7.8.0->black->-r requirements.txt (line 5)) (3.0.38)\n",
            "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.9/dist-packages (from ipython>=7.8.0->black->-r requirements.txt (line 5)) (67.6.1)\n",
            "Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.9/dist-packages (from ipython>=7.8.0->black->-r requirements.txt (line 5)) (0.1.6)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.9/dist-packages (from ipython>=7.8.0->black->-r requirements.txt (line 5)) (0.7.5)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.9/dist-packages (from ipython>=7.8.0->black->-r requirements.txt (line 5)) (2.14.0)\n",
            "Collecting jedi>=0.16\n",
            "  Downloading jedi-0.18.2-py2.py3-none-any.whl (1.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m89.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: backcall in /usr/local/lib/python3.9/dist-packages (from ipython>=7.8.0->black->-r requirements.txt (line 5)) (0.2.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.9/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio->-r requirements.txt (line 12)) (0.1.2)\n",
            "Collecting linkify-it-py<3,>=1\n",
            "  Downloading linkify_it_py-2.0.0-py3-none-any.whl (19 kB)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.9/dist-packages (from pandas->datasets->-r requirements.txt (line 7)) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas->datasets->-r requirements.txt (line 7)) (2022.7.1)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests>=2.19.0->datasets->-r requirements.txt (line 7)) (3.4)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests>=2.19.0->datasets->-r requirements.txt (line 7)) (1.26.15)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests>=2.19.0->datasets->-r requirements.txt (line 7)) (2022.12.7)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.9/dist-packages (from torch>=1.4.0->accelerate->-r requirements.txt (line 1)) (1.11.1)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.9/dist-packages (from torch>=1.4.0->accelerate->-r requirements.txt (line 1)) (2.0.0)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.9/dist-packages (from torch>=1.4.0->accelerate->-r requirements.txt (line 1)) (3.0)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.9/dist-packages (from triton==2.0.0->torch>=1.4.0->accelerate->-r requirements.txt (line 1)) (16.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.9/dist-packages (from triton==2.0.0->torch>=1.4.0->accelerate->-r requirements.txt (line 1)) (3.25.2)\n",
            "Collecting starlette<0.27.0,>=0.26.1\n",
            "  Downloading starlette-0.26.1-py3-none-any.whl (66 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m66.9/66.9 KB\u001b[0m \u001b[31m9.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sniffio in /usr/local/lib/python3.9/dist-packages (from httpx->gradio->-r requirements.txt (line 12)) (1.3.0)\n",
            "Collecting httpcore<0.17.0,>=0.15.0\n",
            "  Downloading httpcore-0.16.3-py3-none-any.whl (69 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m69.6/69.6 KB\u001b[0m \u001b[31m10.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting rfc3986[idna2008]<2,>=1.3\n",
            "  Downloading rfc3986-1.5.0-py2.py3-none-any.whl (31 kB)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio->-r requirements.txt (line 12)) (1.0.7)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio->-r requirements.txt (line 12)) (3.0.9)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio->-r requirements.txt (line 12)) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio->-r requirements.txt (line 12)) (1.4.4)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio->-r requirements.txt (line 12)) (4.39.3)\n",
            "Requirement already satisfied: importlib-resources>=3.2.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio->-r requirements.txt (line 12)) (5.12.0)\n",
            "Collecting h11>=0.8\n",
            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 KB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: anyio<5.0,>=3.0 in /usr/local/lib/python3.9/dist-packages (from httpcore<0.17.0,>=0.15.0->httpx->gradio->-r requirements.txt (line 12)) (3.6.2)\n",
            "Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.9/dist-packages (from importlib-resources>=3.2.0->matplotlib->gradio->-r requirements.txt (line 12)) (3.15.0)\n",
            "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.9/dist-packages (from jedi>=0.16->ipython>=7.8.0->black->-r requirements.txt (line 5)) (0.8.3)\n",
            "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema>=3.0->altair>=4.2.0->gradio->-r requirements.txt (line 12)) (0.19.3)\n",
            "Collecting uc-micro-py\n",
            "  Downloading uc_micro_py-1.0.1-py3-none-any.whl (6.2 kB)\n",
            "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.9/dist-packages (from pexpect>4.3->ipython>=7.8.0->black->-r requirements.txt (line 5)) (0.7.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.9/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=7.8.0->black->-r requirements.txt (line 5)) (0.2.6)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.9/dist-packages (from sympy->torch>=1.4.0->accelerate->-r requirements.txt (line 1)) (1.3.0)\n",
            "Building wheels for collected packages: fire, peft, transformers, ffmpy\n",
            "  Building wheel for fire (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for fire: filename=fire-0.5.0-py2.py3-none-any.whl size=116952 sha256=d9966a17cd65b64d2bf2858a0d2c1827917d137ac645ede196a70eba4223d175\n",
            "  Stored in directory: /root/.cache/pip/wheels/f7/f1/89/b9ea2bf8f80ec027a88fef1d354b3816b4d3d29530988972f6\n",
            "  Building wheel for peft (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for peft: filename=peft-0.3.0.dev0-py3-none-any.whl size=49855 sha256=32163ee65809f051ba3e1fda9ee3b6f3f779a97ef9a3acac4991b2580b9379d6\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-8ty6xkkw/wheels/2d/60/1b/0edd9dc0f0c489738b1166bc1b0b560ee368f7721f89d06e3a\n",
            "  Building wheel for transformers (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for transformers: filename=transformers-4.28.0.dev0-py3-none-any.whl size=6895957 sha256=c82892d2583be352cbe4d5f1aa2e4d88d6ca0363239a644baf792edd944cd09c\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-8ty6xkkw/wheels/f7/92/8c/752ff3bfcd3439805d8bbf641614da38ef3226e127ebea86ee\n",
            "  Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ffmpy: filename=ffmpy-0.3.0-py3-none-any.whl size=4707 sha256=07a193b603fec97362baf967d1a3dcbfdb3ef1c0a7a9fa14f599a79428efafa3\n",
            "  Stored in directory: /root/.cache/pip/wheels/91/e2/96/f676aa08bfd789328c6576cd0f1fde4a3d686703bb0c247697\n",
            "Successfully built fire peft transformers ffmpy\n",
            "Installing collected packages: tokenizers, sentencepiece, rfc3986, pydub, ffmpy, bitsandbytes, xxhash, websockets, uc-micro-py, tokenize-rt, semantic-version, python-multipart, pathspec, orjson, mypy-extensions, multidict, loralib, jedi, h11, frozenlist, fire, dill, async-timeout, aiofiles, yarl, uvicorn, starlette, responses, multiprocess, mdit-py-plugins, linkify-it-py, huggingface-hub, httpcore, black, aiosignal, transformers, httpx, gradio-client, fastapi, aiohttp, gradio, datasets, accelerate, peft\n",
            "Successfully installed accelerate-0.18.0 aiofiles-23.1.0 aiohttp-3.8.4 aiosignal-1.3.1 async-timeout-4.0.2 bitsandbytes-0.37.2 black-23.3.0 datasets-2.11.0 dill-0.3.6 fastapi-0.95.0 ffmpy-0.3.0 fire-0.5.0 frozenlist-1.3.3 gradio-3.24.1 gradio-client-0.0.8 h11-0.14.0 httpcore-0.16.3 httpx-0.23.3 huggingface-hub-0.13.4 jedi-0.18.2 linkify-it-py-2.0.0 loralib-0.1.1 mdit-py-plugins-0.3.3 multidict-6.0.4 multiprocess-0.70.14 mypy-extensions-1.0.0 orjson-3.8.10 pathspec-0.11.1 peft-0.3.0.dev0 pydub-0.25.1 python-multipart-0.0.6 responses-0.18.0 rfc3986-1.5.0 semantic-version-2.10.0 sentencepiece-0.1.97 starlette-0.26.1 tokenize-rt-5.0.0 tokenizers-0.13.3 transformers-4.28.0.dev0 uc-micro-py-1.0.1 uvicorn-0.21.1 websockets-11.0.1 xxhash-3.2.0 yarl-1.8.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Run the application"
      ],
      "metadata": {
        "id": "Cr3bQkSePfrG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Choose models\n",
        "\n",
        "base_model = 'decapoda-research/llama-7b-hf' #@param [\"decapoda-research/llama-7b-hf\", \"decapoda-research/llama-13b-hf\", \"decapoda-research/llama-30b-hf\"]\n",
        "finetuned_model = 'wangrongsheng/chatgentitle-lora-cs-6-new' #@param [\"wangrongsheng/chatgentitle-lora-cs-6-new\", \"wangrongsheng/chatgentitle-lora-all-3-new\", \"wangrongsheng/chatgentitle-lora-cs-3-new\"]\n"
      ],
      "metadata": {
        "id": "4Wg0eqnkPnq-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Run the application\n",
        "\n",
        "It will take some time since LLaMA weights are huge. \n",
        "\n",
        "Click the URL appeared in the `Running on public URL:` field from the log. That will bring you to a new browser tab which opens up the running application."
      ],
      "metadata": {
        "id": "b81jhdtcQyOP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python generate.py --load_8bit --base_model $base_model --lora_weights $finetuned_model --share_gradio"
      ],
      "metadata": {
        "id": "y3qpzBw2jMHq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Have fun!"
      ],
      "metadata": {
        "id": "mc5WIyxYGDo5"
      }
    }
  ]
}